<html>
<head><meta charset="utf-8"><title>fuzzing the compiler · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html">fuzzing the compiler</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="179428578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179428578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179428578">(Oct 30 2019 at 13:03)</a>:</h4>
<p>So <span class="user-mention" data-user-id="247084">@Siavosh Zarrasvand</span> opened <a href="https://github.com/rust-lang/compiler-team/issues/212" target="_blank" title="https://github.com/rust-lang/compiler-team/issues/212">compiler-team#212</a>, a proposal to fuzz the rust compiler and stdlib:</p>
<blockquote>
<p>Creating a OSS Fuzz project for Rust Compiler and stdlib <a href="https://github.com/rust-lang/compiler-team/issues/212" target="_blank" title="https://github.com/rust-lang/compiler-team/issues/212">compiler-team#212</a> </p>
</blockquote>
<p>I'm opening this topic to discuss the idea a bit. Some of the questions I wanted to ask:</p>
<ul>
<li>can we direct the fuzzing in particular ways? <span class="user-mention" data-user-id="116083">@pnkfelix</span> and I were talking about the idea of using fuzzing to target and test incremental. This is something I think <span class="user-mention" data-user-id="124287">@mw</span> and I talked about but  we never did, as far as I know.</li>
<li>if we were to do this, are we concerned about the number of bug reports we'll get? I guess they might serve as an interesting ICE-breaker topic as well, since I expect many to be minor.</li>
<li>any other related thoughts?</li>
</ul>



<a name="179428721"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179428721" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179428721">(Oct 30 2019 at 13:05)</a>:</h4>
<blockquote>
<ul>
<li>if we were to do this, are we concerned about the number of bug reports we'll get?</li>
</ul>
</blockquote>
<p>Hmm.  I hadn't thought of this; it does seem like a potential headache.</p>



<a name="179428756"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179428756" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179428756">(Oct 30 2019 at 13:05)</a>:</h4>
<p>we don't really have any great methodology in place for identify duplicate bugs</p>



<a name="179428781"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179428781" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179428781">(Oct 30 2019 at 13:05)</a>:</h4>
<p>(and to be fair, I don't even know what the state of the art is in that area; as far as I know its still open research to correlate such things...)</p>



<a name="179428932"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179428932" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179428932">(Oct 30 2019 at 13:07)</a>:</h4>
<p>It seems non-useful, i.e., if we expected soundness bugs to be uncovered, then that seems useful</p>



<a name="179429008"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429008" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429008">(Oct 30 2019 at 13:08)</a>:</h4>
<p>but if it's just finding ICEs -- well, we have 268 of those filed right now :)</p>



<a name="179429023"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429023" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429023">(Oct 30 2019 at 13:08)</a>:</h4>
<p>so finding more doesn't seem too useful</p>



<a name="179429063"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429063" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429063">(Oct 30 2019 at 13:08)</a>:</h4>
<p>we can certainly categorize which kinds of bugs are worth filing</p>



<a name="179429213"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429213" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429213">(Oct 30 2019 at 13:10)</a>:</h4>
<p>also: an ICE exposed solely by incremental, that has a <em>reprodicible</em> test case, <em>is</em> useful</p>



<a name="179429268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429268">(Oct 30 2019 at 13:10)</a>:</h4>
<p>that is, I think we have a number of ICE's filed that are associated with incremental, but the bug filers often do not have a reproducible test case on hand, because they are not certain <em>why</em> it started failing, and <code>cargo clean</code> makes the bug go away</p>



<a name="179429278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429278">(Oct 30 2019 at 13:11)</a>:</h4>
<p>and those bugs are often essentially non-actionable.</p>



<a name="179429311"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429311" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429311">(Oct 30 2019 at 13:11)</a>:</h4>
<p>So my claim is that fuzzing might provide a way to get easy to reproduce bugs in incremental-compilation</p>



<a name="179429456"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429456" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429456">(Oct 30 2019 at 13:12)</a>:</h4>
<p>oh, yes, definitely true</p>



<a name="179429460"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429460" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429460">(Oct 30 2019 at 13:13)</a>:</h4>
<p>(also, it would just give me more faith that we are exercising more corner cases for incremental compilation testing. Right now I am scared to close the bugs that do not have non-reproducible test cases, because I don't want to lose the record that a latent bug <em>is</em> still believed to be present ...)</p>



<a name="179429484"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429484" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429484">(Oct 30 2019 at 13:13)</a>:</h4>
<p>my impression was that fuzzing is not testing incremental? If we can do that, then amazing!</p>



<a name="179429497"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429497" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429497">(Oct 30 2019 at 13:13)</a>:</h4>
<p>My idea that I was saying to Niko was to use fuzzing as a way to test incremental</p>



<a name="179429499"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429499" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429499">(Oct 30 2019 at 13:13)</a>:</h4>
<p>Fuzzing is nice, but it's a fairly blunt instrument; sometime we should start using proper property based testing as well</p>



<a name="179429503"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429503" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429503">(Oct 30 2019 at 13:13)</a>:</h4>
<p>one crazy thought: run the ui test suite in incremental mode with a single directory</p>



<a name="179429517"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429517" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429517">(Oct 30 2019 at 13:13)</a>:</h4>
<p>i.e. compile, change a bit, compile incrementally, then clean, then compile from scratch</p>



<a name="179429567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429567">(Oct 30 2019 at 13:14)</a>:</h4>
<p>and compare the latter two compilation sessions</p>



<a name="179429592"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429592" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429592">(Oct 30 2019 at 13:14)</a>:</h4>
<p>because, ideally, they would have the same diagnostic output, if any</p>



<a name="179429598"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429598" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429598">(Oct 30 2019 at 13:14)</a>:</h4>
<p>yes, that's true</p>



<a name="179429641"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429641" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429641">(Oct 30 2019 at 13:14)</a>:</h4>
<blockquote>
<p>Fuzzing is nice, but it's a fairly blunt instrument; sometime we should start using proper property based testing as well</p>
</blockquote>
<p>I am a bit confused -- these seem pretty equivalent to me? In the sense that our property is "equivalent whether using incremental or not"</p>



<a name="179429642"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429642" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429642">(Oct 30 2019 at 13:14)</a>:</h4>
<blockquote>
<p>one crazy thought: run the ui test suite in incremental mode with a single directory</p>
</blockquote>
<p>can you elaborate on this? Like, every test input would use the same shared incremental store?</p>



<a name="179429656"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429656" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429656">(Oct 30 2019 at 13:15)</a>:</h4>
<p>Right, yeah</p>



<a name="179429672"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429672" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429672">(Oct 30 2019 at 13:15)</a>:</h4>
<p>obviously this would mean -j1 or so</p>



<a name="179429692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429692">(Oct 30 2019 at 13:15)</a>:</h4>
<p>but we would expect that incremental does not "hide" anything</p>



<a name="179429699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429699">(Oct 30 2019 at 13:15)</a>:</h4>
<p>sure. Is part of the idea that many of our tests have similar filenames etc ?</p>



<a name="179429725"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429725" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429725">(Oct 30 2019 at 13:15)</a>:</h4>
<p>(i'm just trying to understand what corner case this is expecting to tickle)</p>



<a name="179429741"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429741" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429741">(Oct 30 2019 at 13:16)</a>:</h4>
<p>oh, I was thinking we rename them to <a href="http://foo.rs" target="_blank" title="http://foo.rs">foo.rs</a> or w/e, not different names</p>



<a name="179429789"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429789" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429789">(Oct 30 2019 at 13:16)</a>:</h4>
<p>this is just an easy corpus for which we have pre-existing output</p>



<a name="179429797"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429797" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429797">(Oct 30 2019 at 13:16)</a>:</h4>
<p>gotcha</p>



<a name="179429856"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429856" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429856">(Oct 30 2019 at 13:17)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> With property based testing in terms of quickcheck and proptest you define how to generate data types and how to shrink them yourself</p>



<a name="179429912"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179429912" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179429912">(Oct 30 2019 at 13:17)</a>:</h4>
<p>hm well okay, I guess "better" in some sense, but in our case I have no _good_ idea how one would do that. I guess maybe taking wg-grammar's grammar and generate output based on that</p>



<a name="179430025"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430025" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430025">(Oct 30 2019 at 13:18)</a>:</h4>
<p>basically yes, for some syntax -- but you can also property test some smaller parts of the compiler or just the standard library</p>



<a name="179430058"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430058" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430058">(Oct 30 2019 at 13:19)</a>:</h4>
<p>I agree that proptesting is good</p>



<a name="179430079"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430079" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430079">(Oct 30 2019 at 13:19)</a>:</h4>
<p>one can also generate ASTs directly and not take a syntax detour</p>



<a name="179430092"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430092" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430092">(Oct 30 2019 at 13:19)</a>:</h4>
<p>but the simplicity of fuzzing does make it an attractive option, just in terms of having its infrastructure be super super simple</p>



<a name="179430111"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430111" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430111">(Oct 30 2019 at 13:19)</a>:</h4>
<p>(and thus easy for purposes of onboarding contributor)</p>



<a name="179430183"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430183" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430183">(Oct 30 2019 at 13:20)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> I think they can be complementary</p>



<a name="179430195"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430195" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430195">(Oct 30 2019 at 13:20)</a>:</h4>
<p>I agree with the complementary nature</p>



<a name="179430198"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430198" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430198">(Oct 30 2019 at 13:20)</a>:</h4>
<p>and they need not stand in each other's way</p>



<a name="179430368"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430368" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430368">(Oct 30 2019 at 13:22)</a>:</h4>
<p>(it's worth noting that the proposal also describes libstd; although that's not our "balliwick", it could definitely be useful)</p>



<a name="179430499"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430499" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430499">(Oct 30 2019 at 13:23)</a>:</h4>
<p>(I think t-compiler should be empowered to improve libstd when it doesn't change the spec; much like t-compiler can improve the language when it doesn't change the spec)</p>



<a name="179430680"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430680" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430680">(Oct 30 2019 at 13:25)</a>:</h4>
<p>... anyone's welcome to contribute changes  (to libstd). (or at least propose PRs thereof.) If you're talking about having power to <em>approve</em> changes, I don't see the need nor want the responsibility</p>



<a name="179430841"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430841" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430841">(Oct 30 2019 at 13:27)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span>  if you have r+ you can even add new unstable APIs (with T-libs blessing) if you think it's reasonable and small</p>



<a name="179430852"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430852" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430852">(Oct 30 2019 at 13:27)</a>:</h4>
<p>(anyways... different subject)</p>



<a name="179430986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430986">(Oct 30 2019 at 13:28)</a>:</h4>
<blockquote>
<p>(I think t-compiler should be empowered to improve libstd when it doesn't change the spec; much like t-compiler can improve the language when it doesn't change the spec)</p>
</blockquote>
<p>obviously. I'm just saying that maybe part of the answer is: let's try this out, but focused on libs</p>



<a name="179430992"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179430992" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179430992">(Oct 30 2019 at 13:28)</a>:</h4>
<p>though I personally think the incremental fuzzing is a good ide</p>



<a name="179431031"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179431031" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179431031">(Oct 30 2019 at 13:29)</a>:</h4>
<p>I'm not sure, maybe <span class="user-mention" data-user-id="247084">@Siavosh Zarrasvand</span> can clarify, what sort of fuzzing they had in mind -- i.e., we don't have to necessarily just mutate random bytes, we can do more structured changes</p>



<a name="179431037"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179431037" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179431037">(Oct 30 2019 at 13:29)</a>:</h4>
<p>for example, we might try renaming variables</p>



<a name="179431059"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179431059" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179431059">(Oct 30 2019 at 13:29)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> I think that's a great idea; running fuzzing &amp; PBT on the standard library should be way easier</p>



<a name="179431064"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179431064" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179431064">(Oct 30 2019 at 13:29)</a>:</h4>
<p>or other things that we expect to be "perfect" in some sense</p>



<a name="179431104"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179431104" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179431104">(Oct 30 2019 at 13:30)</a>:</h4>
<p>especially for PBT, you can define some small properties for some algorithmic parts like iterators</p>



<a name="179431216"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179431216" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> centril <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179431216">(Oct 30 2019 at 13:30)</a>:</h4>
<p>Maybe worth asking GHC what they do -- I have a feeling they use a lot of QuickCheck for stuff</p>



<a name="179431587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/fuzzing%20the%20compiler/near/179431587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Siavosh Zarrasvand <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/fuzzing.20the.20compiler.html#179431587">(Oct 30 2019 at 13:34)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> Yes, especially letting the fuzzer alter bits on a working seed input file will probably never allow it to get past the lexer. <br>
Fuzzers can usually be programmed to alter only specific parts of the input file using string matching. We could start with very basic <a href="http://main.rs" target="_blank" title="http://main.rs">main.rs</a> files and fuzz the syntax initially (see this approach: <a href="https://github.com/vegard/prog-fuzz" target="_blank" title="https://github.com/vegard/prog-fuzz">https://github.com/vegard/prog-fuzz</a>) and increase coverage to std library modules long term. I was hoping to start very small and build incrementally. </p>
<p>Regarding bug concerns, the initial bugs on the compiler will have near 0 severity. They will crash rustc with input that it doesn't know how to lex. Later on in the std libraries, issues could have some severity...</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>